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The  DevOps  Movement  Began  as  a  Reaction  .. 


To  years  of  disconnect  between  Dev  and  Ops  which  began  to 
manifest  itself  as  conflict  and  inefficiency 
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Familiar  DevOps  Problems 


•  Disconnect  between  Dev  and  Ops  teams  leads  to  a  wall  of  confusion 
between  stove-piped  teams 

•  Disconnects  between  Dev  and  Ops  tools,  as  well  as  processes, 
cause  inefficiency  and  rework 


Development 


Operations 


J  Ops  Tools  - 


Source:  Lee  Thompson  and  Andrew  Shaffer 
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DevOps  is  helping  to  finish  what  Agile  started 


We  saw  reduced 
development  cycle  time 
with  Agile,  but  due  to 
issues  such  as: 

•  Lack  of  confidence  in 
deployment/  rollback 

•  Inefficient  test 
approaches,  etc. 

•  Unreliable  software 

Deployment  cycle  time 
is  often  weeks  or  months 
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No  Value  gained  when  Software  is  not  Delivered 


Software  Engineering  Institute  Carnegie  Mellon  University 


Software  Architecture: 

Trends  and  New  Directions 
#SEIswArch 

©  2014  Carnegie  Mellon  University 


Informal  DevOps  Definitions 


“DevOps  is  a  software  development  method  that 
stresses  communication,  collaboration  and  integration 
between  software  developers  and  information 
technology  (IT)  professionals” 

Pant,  Rajiv 


“DevOps  is  an  umbrella  concept  for  anything  that 
smooth's  out  the  interaction  between  development 

and  operations” 

Damon  Edwards 
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Scope 


The  scope  for  DevOps  looks  at  reducing  deployment  cycle  time  and 
enabling  feedback  cycles  across  the  end-to-end  Deployment  Pipeline  .. 
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Challenges  DevOps  is  trying  to  Soive 


r 
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r 
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Lost  structural 
information 


Lost  behavioral 
information 


•  Non-collaborative  stove-piped  Dev  and  Ops  teams 

•  Limited  improvement  within  stove-piped  areas  (e.g.,  process,  tools, 
metrics)  but  not  end-to-end 

•  Broken  feedback  cycles;  process  flows  only  one  way 


Forrester,  The  Seven  Habits  Of  Highly  Effective  DevOps 
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DevOps  Community  Future  Vision 


Agile  process 


Feedback 


Preserved  structural 
information 


Preserved  behavioral 
information 


Feed-forward 


•  Collaborative,  Dev  and  Ops  teams  combine  or  working  closely  together 

•  Continuous  improvement  across  the  deployment  pipeline  targeted  at  producing 
something  of  value  to  a  user  or  organization  (inception  to  dev  to  release/sustain) 

•  Effective  feedback  cycles  within  each  stage 


Adapted  from  Forrester,  The  Seven  Habits  Of  Highly  Effective  DevOps 
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More  than  Dev  and  Ops  Working  Together 


Those  are  some  of  the  overarching  goals  of  DevOps,  but  is  easy  to  think 
of  DevOps  as  just  a  collaborative  movement  because  people  get  that 


•  •  •  •  • 


TEAMWORK 

wiifiN 


But  it  is  really  more  than  that 
•  There  are  multiple  dimensions  to  the  movement. . . 
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Multiple  Dimensions  of  DevOps 


Automation/ 

Measurement 


System/ 

Architecture 


Culture 


Culture 


Developer  and  Ops  collaboration 
(Ops  includes  Security) 

Developers  and  Operations  support 
releases  beyond  deployment 

Dev  and  Ops  have  access  to 
stakeholders  who  understand 
business  and  mission  goals 


Automation/ 

Measurement 

•  Automate  repetitive  and 
error-prone  tasks  (e.g.,  build, 
testing,  deployment,  maintain 
consistent  environments) 

•  Static  analysis  automation 
(architecture  health) 

•  Performance  dashboards 


Process  and 
Practices 


Pipeline  streamlining 

Continuous  Delivery 
practices  (e.g. 
Continuous  Integration, 
Test  Automation,  Script- 
driven,  automated 
deployment.  Virtualized, 
self-service 
environments) 


System/Architecture 


Architected  to  support  test 
automation  and  continuous 
integration  goals 


Applications  that  support 
changes  without  release  (e.g., 
late  binding) 


Scalable,  secure,  reliable,  etc. 


Ignoring  any  of  these  dimensions  can  cause  problems 
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Feedback  Cycle  Breakdown  Examples 


Architecture  can  enable  or  imped  short  feedback  cycle  time 


Deployment  Pipeline 


Examples  of  Feedback  Cycle  breakdown  due  to  Architecture  Issues: 

•  F1 :  Builds  take  too  long  due  to  poorly  managed  component  dependencies:  integration  builds 
are  slow  and  become  infrequent 

•  F2:  System  doesn’t  have  architectural  interfaces  for  test  automation  and  manual  tests  are 
slow;  tests  are  skipped 

•  F3a&b:  Architecture  creates  deployment  complexity  and  error  prone  manual  steps  prevent 
release:  weeks/months  without  release 
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Challenge  Questions 


We  just  gave  several  examples  of  how  architecture  can  enable  or 
impede  feedback  cycles,  and  consequently,  end-to-end  deployment 
cycle  time  (we  refer  to  as  Deployability) 

However,  this  raises  several  questions  such  as: 

•  How  do  we  specify  Deployability  requirements  clearly  and  concisely? 

•  How  do  we  design  systems  for  Deployability? 

•  What  kinds  of  design  decisions  really  matter? 

•  Are  there  architectural  tactics  and/or  patterns  we  might  want  to  leverage  to  promote 
Deployability? 

•  When  planning  work,  what  Deployability-related  requirements  and  design 
decisions  should  be  considered  early  to  avoid  rework? 
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Requirements  for  Deployability 


Lack  clear  specification  for  Depioyabiiity  requirements  ieads  to  feedback 
cycie  breakdowns 

Exampie  Vague  Requirements: 

“Our  system,  and  delivery  environment,  shall  support  continuous 
delivery  and  multiple  deploys  a  day  like  Amazon,  Google,  etc." 


"When  it  comes  to  deployment,  everything  possible  should  be 
automated" 

In  next  few  slides,  we  give  examples  of  Deployability  requirements  that 
enable  better  feedback  across  the  deployment  pipeline 
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Specifying  Depioyabiiity  Requirements 


Well  specified  requirements  enable  Feedback  Cycles;  Several 
example  Deployablity  Requirements  are  shown  below: 

P1:  Build  and  Continuously  Integrate 

•  Complete  full  software  build  in  <  5  minutes  under  peak  load 

P2:  Automated  Testing 

•  Complete  execution  of  Unit  tests  suite  within  10  minutes 

•  Complete  execution  of  increment  tests  suite  (e.g.,  NFR)  within  5  hours 

•  Create/build  a  new  system-level  test  case,  avg  time  to  build/test  is  1  day 

P3:  Automated  Release 

•  There  is  an  upgrade  being  pushed  out,  99%  of  release  is  automated 
and  1%  is  handled  manually 

•  The  team  makes  a  change  to  feature  X  (Ul  and  business  logic  change) 
and  deploy  is  pushed  out  within  2  hours  of  code/test  completion 

Source:  ATAM  Analysis  Data  2006-2013 
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Requirements  Mapped  to  Feedback  Cycles 


Deployablity  requirements  specified  as  quality  attributes  can  provide 
concrete  measures  for  designing  systems  to  achieve  feedback  cycle  time 


Deployment  Pipeline 


Source:  Towards  Design  Decisions  to  Enable  Deployability,  DSSO  workshop  paper  submission  (in  review) 
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Design 


to  promote  Deployability 


•  We  just  gave  examples  of  Deployability  requirements;  next  we 
investigate  design  decisions.  We  draw  upon  interviews  with  projects 
practicing  continuous  delivery  (sampling  below)... 


Project 

Management 

Approach 

Size  Metrics 

Years 

In  Use 

Release  Cadence 

Cl 

Cadence 

A 

Agile/Scrum 
(last  2  years 
and  traditional 
before  that) 

1M  SLOG 

17 

Client  release 
available  every  2 
months  (not  all  accept 
it) 

Daily  Cl 
build 

B 

Water/Scrum/F 

all 

3M  SLOG, 
team  size  6- 
8, 

90,000  users 

3+ 

Internal  release  every 
2-3  weeks,  external 
release  as  needed 

Daily  Cl 
build 

C 

Agile/Scrum 

Team  size  30 

2+ 

Internal  release  every 
2-3  weeks,  customer 
release  every  2-3 
months 

Daily  Cl 
build 

Source:  Towards  Design  Decisions  to  Enabie  Depioyability,  submitted  Dependability  and  Security  Workshop, 
Beiiomo,  Kazman,  Ernst 
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Architecture  Partitioning  Decision 

Decision:  Divide  components  and  allocation  teams 
separately  to  promote  rapid  builds  and  tests 

•  Changes  to  blue  components  (Team  B)  do  not  require 

rebuild  of  yellow  components  (Team  A)  which  shortens  build  time 


Trade-offs 

+Modifiability 
+T0Stability 
+Reduced  Build  Time 
-Reuse 
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Integrated  Test  Harness  Decision 


Decision:  Integrate  test  harness  hooks  to 

architecture  to  start  and  stop  application 

(start  in  clean  state,  end  test  with  clean  environment) 

•  Shortened  Test  Duration 


Trade-offs 

+Testability 

+Modifiability 

-Complexity 


Test  execution 
engine 
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Start/Stop 

Application  1 


Application  2 


Legacy 

Component 


Legacy 

interfaces  may 
need  to  be 
refactored  for 
automation 
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Web  Services  Layer  Removal  Decision-1  a 


Decision:  Remove  web 
services  layer;  replace 
with  Enterprise  Java  Bean 
implementation 

•  Minimized  Deployment 
complexity 


Trade-offs 

+Releasabillty 

+Reduced 

Complexity 

+Performance 

-Testability 

-Modifiability 


Module  View 


Web  App  (Java 
Server  Faces) 


Before 


Middletier  (J2EE) 


Component 

Implementation 


EJBs 
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Web  Services  Layer  Removal  Decision-1  b 
(Before  redesign) 


— iLJ 

Firewall 


Web  App 
Release 
package 
1.2 

Web  App 
Release 
package 
1.2 


(cold2} 


Web  Server 
(hotl) 


<- 


Web 

Services 

Server 


Web  Server 
{hot2) 


Web 

Services 

Release 

package 

2.3 


Web 

Services 

Release 

package 

2.3 


Before,  had  to  update  multiple  application  servers  and  web  services 
to  be  sure  that  application  and  services  versions  were  in  synch 
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Web  Services  Layer  Removal  Decision-1  c 
(After  redesign) 


"After" 


Load 

Balancer 


Firewall 

Proxy  Web 
Server 


Web/Server 

MiddleTier 

Server 

(coldl) 


Web/Server 
MiddleTier 
Server  {cold2) 


Web/Server 
MiddleTier 
Server  (hotl) 


Web/Server" 
d  MiddleTier 
Server  (hot2) 


Release 

package 

1.2 


Release 

package 

1.2 


Release 

package 

1.2 


Release 

package 
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Benefits 
■  All  boxes 
configured 
the  same 

*  App 
pushed 
once  and 
runs  on 
cold  and 
hot  boxes 
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Web  Service  Consolidation  Decision 


Decision  Example:  Consolidate  Web  Services  for  easier  release, 
increased  performance  and  reduced  complexity 


Before 


Application 


After 


Application 


Trade-offs 

+Releasability 

+Reduced  Deploy 

Complexity 

+Performance 

-Testability 

-Modifiability 
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Mapping  Design  Decisions  to  Pipeiine 

Each  design  decision  also  supports  the  pipeline  feedback  loops 


Deployment  Pipeline 


Decision  Decision  Decision  Decision 

Source:  Towards  Design  Decisions  to  Enabie  Depioyability,  DSSO  workshop  paper  submission  (in  review) 
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Relating  Terms  and  Concepts 

In  the  next  few  slides,  we  give  a  few  examples  that  connect  from 
requirements  to  design  decisions  to  tactics;  The  ER  diagram  below 
provides  an  overview  of  concepts  we  are  discussing 


Problem  space  Solution  space 


Integrated  Test  Harness  Example 


Problem:  Long  testing  duration  due  to  problems  with  establishing 
clean  test  start  state  and  difficulty  executing  tests  In  automated 
fashion  (manual  steps  required) 


Broken  Feedback  loop: 

Long  Automated  Testing  Cycle 


Requirement 

Scenario: 

“Complete  execution 
of  increment  tests 
suite  (e.g.,  NFR) 
within  5  hours” 


Design 
Decision: 
Integrated 
test  harness 


Fixed  Feedback  loop: 
Shortened  Test  Duration 


0 


Tactics  Used: 

Specialized  Access 
Routines 
Record/playback 
Maintain  Interfaces, 
State  Synchronization  & 
resychronization 
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Modular  and  Distributed  Architecture  Exampie 


Problem:  Long  deployment  duration  due  to  problems  with 
architectural  dependencies 


Broken  Feedback  loop:  Fixed  Feedback  loop: 

Infrequent  deployments  Reduced  Deployment  time 


^ 


Requirement  Scenario: 

Tactics  Used: 

“The  team  makes  a 

Design 

•  Increase  Semantic 

change  to  feature  X  (Ul 

1— K.  Decision: 

Coherence 

and  business  logic 

Distribute  (SiH/ 

•  Encapsulation 

change)  and  deploy  is 

modularixe 

•  Maintain  Existing 

pushed  out  within  2  hours 

architecture 

Interfaces 

of  code/test  completion” 

“If  you  push  the  whole  three  million  line  appiication  every  time  a  change  is 

made  you  are  in  a  world  of  hurt”  Project  C 
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Deployability  Architecture  Tactics  Tree 


Top  Row 
Represents 
Stakeholder 
Design  Drivers 


Deployability 

Tactics 


Deployability  Tactics  Summary 


Continuous 
Integration  (Gl) 

Encapsulate 

Increase 

Semantic 

Coherence 

Maintain  Existing 
Interfaces 


Automation 

(G2) 


Deployment  and  Robust 
Operations  (63} 


Specialized  Access 
Routines 

Record/Playback 

(similarto) 

Encapsulate 

State 

Synchronization  and 
Resynchronization 


These  tactics  also 
enable  test  automation 


Monitor 

Exception  Detection 
Exception  Handling 
Abstract  Common 
Services 

Generalize  Module 
Maintain  Multiple 
Copies 

Increase  Available 
Resources 

Increase 

Computational 

Efficiency 

Reduce  Overhead 


Flexible 

Environments  (64) 

Defer  Binding 
Sandbox  (similarto) 
Virtualization 
Condition  Monitoring 
Active  Redundancy 
Rollback 


Source:  Towards  Design  Decisions  to  Enable  Deployability, 

submitted  Dependability  and  Security  Workshop,  Bellomo,  Kazman,  Ernst 
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Modular  and 
Distributed 
Architecture 
Exampie 


Depfoyability  Tactics  Summary 


Enable 
Continuous 
Integration  (61) 

Encapsulate 

Increase 

Semantic 

Coherence 

Maintain  Existing 
Interfaces 


Integrated 
Test  Harness 
Example 


Enable  Test 
Automation 
(62) 


Enable  Rapid 
Deployment  and  Robust 
Operations  (63) 


Specialized  Access 
Routines 

Record/Playback 
(similar  to) 

Encapsulate 

State 

Synchronization  and 
Resynchronization 


these  tactics  also 
enable  test  automation 


Monitor 

Exception  Detection 
Exception  Handling 
Abstract  Common 
Services 

Generalize  Module 
Maintain  Multiple 
Copies 

Increase  Available 
Resources 

Increase 

Computational 

Efficiency 

Reduce  Overhead 


Enable 

Synchronized  and 
Flexible 

Environments  (64) 

Defer  Binding 
Sandbox  (similarto) 
Virtualization 
Condition  Monitoring 
Active  Redundancy 
Rollback 


‘Need  Speed 
and  Rigor” 


Software  Architecture: 

Trends  and  New  Directions 
#SEIswArch 

©2014  Carnegie  Mellon  University 


Software  Engineering  Institute 


Carnegie  Mellon  University 


Allocating  Deployability 


Our  examples  suggest  some 
Deployablity-related  design 
decisions/trade-offs  can  have 
significant  impact 

In  cases  where  the  structure 
of  the  architecture  is  impacted 
by  a  decision,  it  may  make 
sense  to  consider  them  early 
to  avoid  rework 


State  of 
agile  team 
support 


Desired  State 


Preparation 


Preservation 


Time 


Designing  for  Depioyabiiity,  iike  any  quaiity  attribute,  requires 
weii  informed  architecturai  trade-off  anaiysis 
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Wrap  Up 


In  this  talk,  we  have  shared  an  approach  for: 

•  Describing  Deployability  concerns  as  architecturally  significant  scenarios 

•  Applying  trade-off  analysis  to  make  Deployment-focused  design  decisions 

•  Leveraging  tactics  to  control  Deployability-related  response  measures 

Work  to  be  done 

•  Collect  more  examples  of  scenarios,  design  decisions  and  tactics 

•  Expand  and  further  validate  the  Deployability  tactics  tree 

•  Apply  Deployabliltiy  tactics  to  help  teams  reduce  deployment  cycle  time  and 
enable  feedback  cycles  across  the  deployment  pipeline  (e.g.,  tactic  checklist) 
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Want  to  get  involved? 


Upcoming  activities 

•  IEEE  Software  Magazine  Special  Issue  on  Release  Engineering,  April/May 
2015 

•  SATURN  SEI  Software  Architecture  Conference,  2014,  May  5-9  Portland 
Oregon,  Tutorial  on  Architecture  Tactics  to  Reduce  Deployment  Cycle  Time 

Contact  Information: 


Stephany  Bellomo, 
sbellomo@sei.cmu.edu 

Neil  Ernst 

nernst@sei.cmu.edu 


Rick  Kazman 
kazman@sei.cmu.edu 

Rod  Nord 
rn@sei.cmu.edu 
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